Towards Web Search Engine Scale Data Mining

Data mining is one of the most critical driving technologies behind Web search engines. Web search engine scale data mining posts many grand challenges, ranging from efficiency and scalability to diversity and adaptability. In this talk, I will review our recent effort on mining a very large amount of data accumulated in one of the major commercial search engines. Particularly, we tackle the problem of context–aware search and query suggestion by employing statistical models. Moreover, we construct a very large statistical model (millions of states) from a very large amount of data (billions of sessions) by distributed data mining. I will also introduce some of our recent initiatives in Web mining. Copyright c ©2009, Australian Computer Society, Inc. This paper appeared at the Eighth Australasian Data Mining Conference (AusDM 2009), Melbourne, Australia. Conferences in Research and Practice in Information Technology (CRPIT), Vol. 101, Paul J. Kennedy, Kok-Leong Ong and Peter Christen, Ed. Reproduction for academic, not-for profit purposes permitted provided this text is included. Proc. of the 8th Australasian Data Mining Conference (AusDM'09)